Lines of Latitude & Longitude

Concepts & Themes:

This week’s assignment will encompass the following concepts covered in Class 2 lecture demo & lab:

Visual Analogies for Latitude & Longitude:

A mapped GCS coordinate

Latitude as rungs on ladder

Longitude as telephone poles

Class 2 Readings:

Assignment II Map: Large Scale Mapping of NYC 311 Service Request Dataset

Preamble:

  • In the following assignment steps the NYC’s 311 Service Requests for Covid Social Distancing is utilized as an example dataset. Generally the same steps apply to all 311 themes available. For this assignment, choose a theme and a time period filter for your own assignment submission.

  • In order to choose a theme, you must obviously know which themes are available. In the 311 dataset, themes are organized by Complaint Type first; with subcategories that are listed as Descriptor. A third column is provided for the theme preview titled Unique Key, which in this case is a count of records per Complaint Type + Descriptor. To no one’s surprise, Noise - Residential + Loud Music/Party has the largest count at 1,665,443 records dating back to 2010.

  • To download the theme preview, navigate to, and download:

  • In this assignment, we will map the 311 data, conduct a spatial operation then visualize the results both in cartographic and chart outputs. Often GIS output will feature a combination of aspects, attributes or dimensions of a dataset. A recent example of this approach to data visualization from National Geographic - Monumental Undertaking shows first the temporal dimension of Confederate statues, followed by their spatial distribution:

Example of data presentation - chart + map

  • The following mapping assignment utilizes the spatial distribution of Social Distancing complaints as an assignment example, alongside an aggregate count per NYC borough. By visualizing both the disaggreate data points and aggregate count, the final product allows a meaningful count in addition to spatial distribution alone.

  • To Start, first acquire the data in .csv format, then map it against the 5 boroughs in NYC. The Latitude and Longitude spatial attributes in the dataset will be used to transform the tabular dataset into a spatial dataset with geometry - .csv to .shp. Along the way, a total complaint count per borough will be determined, and finally a rate of complaint per borough. A map product is then developed that shows the pattern of the data and charts the relative complaint count per borough.

  • To start, the following filter query and rate formula will be utilized in the assignment:

    • To filter the 311 data just the Descriptor will be used, set to Social Distancing.
    • In order to determine a square mile per borough:shape_area * 0.00000003587006428 = square mile

Note: for filtering, a combination of Complaint Type and Descriptor can be utilized to narrow results.

Data Acquisition:

NYC OpenData

  • Scroll down to the 311 Dataset link and click:

311 Service Requests

  • Note that there are multiple 311 Service Request instances. Make sure to select the Dataset, not Data Lens instance:

311 Dataset

  • Once selected, continue with View Data button selection:

View Data

  • Apply a Filter based on query Descriptor is Social Distancing:

Applying Data Filter

  • The resulting subselection would be 62,000 + records with the Social Distancing Descriptor attribute:

Record Results

  • Download the subselection .csv to a folder directory for Part II. Accept the default naming convention of the .csv:

Download Record Results

  • If desired, open the .csv in Excel and navigate to right-end of the file. Here you will see Latitude and Longitude. These two columns will be utilized to read in the locational data for the creating of geometry (points) within QGIS:

Locational Data

  • Finally, download the Class2 map data folder. Here you will find the borough geometry, and a backup .csv of the 311 data as of 01/29/21. Only use this .csv as testing data, not data for the assignment per se:

Class 2 Assignment Data

Data Organization within QGIS:

  • Open QGIS and point the Data Source Manager to the .csv service requests utilizing the Delimited Text data source option. Make sure to populate the tool as seen below and rename the layer simply as 311_SR:

Delimited Text - QGIS

  • Next, load the borough boundary shapefile, and position the service requests atop the borough boundary and symbolize so that the the points are approximately 0.25 size in black, and the boroughs are a light neutral color of your choice:

NYC Borough Geography

Note: There will be some stray points outside NYC area. These are incorrect lat/lon data points wherein the lat/lon was recorded incorrectly in the 311 dataset. Disregard these points. Also now is the time to begin to save your .qgs file on a regular basis in order to avoid duplicate work in the event of a software crash.

  • You may notice that your data ‘shape’ is horizontally elongated compared to the image above. This is due to the fact that the data is cast within wgs84 coordinate system. We can change that easily so that the ‘shape’ is much more accurate. To do so, we will utilize the map projection designed for the NYC area (State Plane system). First click on the CRS button, lower right of Map Canvas. Once done, search for 3104 which is a short code for the correct map projection for NYC. Then click Add at bottom of tool window. The shape of the data will automatically update. This is a type of ‘on the fly’ projection; we are changing the Map Canvas coordinates, but we are leaving the actual .prj file on each of the data sources unchanged. This is a ‘quick and dirty’ yet efficient cartographic kludge (hack):

Location of CRS reference - lower right of Map Canvas

CRS Filter

Note: you may run into a CRS projection issue with the polygon feature for NYC boroughs. IF your features disappear from your Map Canvas as you perform an ‘on the fly’ projection for the data frame, the following video will solve this problem:

  • Addressing the Missing CRS delimma:

Lecture 1

  • Next, we want know how many complaints exist per borough expressed in the following structured question:

    • Which borough has the most ‘Social Distancing’ public complaint reports due to Covid-19 distancing mandates in New York City and State ?
    • Which borough has the highest rate of public complaint reports based on square area (square miles) ?

Note: for this second question, perhaps a more insightful query would be rate of public complaints per 10K population. Certainly we can answer this question also, but we would need population data which we will cover in depth in Class 4 - Thematic Mapping.

  • Utilize the Count points in polygon overlay operation to get a total count of complaint records per borough. This tool is available from the Main Menu:

  • Vector > Analysis Tools > Count Points in Polygon

Count Points in Polygon Operation

  • Populate the tool with the polygon and points ordered as follows:

Counts in Polygon Tool Parameters

  • This operation will result in a new temporary layer COUNT. Immediately export this new layer as a .shp into your working directly, entitled 311_borough:

Export Operation

Export Operation as .shp

Note: see that the new .shp now contains a field column entitled NUMPOINTS in integer format (int8) - this is the count per borough.

  • Next, calculate the complaints per borough and the rate of complaints per borough in the attribute table of the new 311_borough.shp. To do this navigate to the Toolbar and choose the Field Calculator tool:

QGIS Field Calculator

  • Make sure 311_borough is active in the layers panel. Create a new field sq.mile and populate with the following formula, using Fields and Values > shape_area dropdown:

    • shape_area * 0.00000003587006428

Field Calculator Operation

Note: This formula takes the current areal quanity in sq. feet and converts it to a sq. mile - a unit that is more appropriate to the size of the borough polygons.

  • To complete, save the attribute addition; Toggle Editing to the off position and Save:

Toggle Editing

  • With the number of complaints per borough (NUMPOINTS) and the sq. miles of each borough (sq.mile), a rate normalized to sq. area can be achieved. Again, open the Field Calculator and use the same process as above to Create a new field, but this time name the field rate.sq.mi and use the following formula:

  • "NUMPOINTS" / "sq.mile"

  • Again, save the attribute addition; Toggle Editing to the off position and Save.

  • Open the attribute table and view the results. Its clear the rate of complaints per sq. mile in Manhattan is highest, even as its total count is lower than Brooklyn. The influence of the outer neighborhoods in Brooklyn is likely bringing down the rate in Brooklyn as a whole. In other words, the most populated areas of Brooklyn are likely on par with the rate of complaints per sq. mile as Manhattan.

Attribute Table with Added Fields

Note: if you are having problems with the Field Calculator at this stage of the assignment, the following video will walk through each step and may be helpful to get back on track:

  • Creating Sq Miles & Rates with QGIS Field Calculator:

area and rates

Visualizing the analysis results - cartographic and graphing design:

  • With the analysis steps complete, we move along to cartographic design and output. We can augment the map design with a graphing feature to good effect. The following components will be featured in this final stage:

    • Utilize QGIS Plugin architecture.
    • Produce cartographic output; augment with graph design.
    • Incorporate HTML elements into the Map Layout.

  • To start, make sure the .qgs files is saved.

Note: as there will be 60K points on this map, consider ramping down opacity and keep small symbol size. For the example map, opacity of points has been set at approx. 75%, and the size 0.25.

  • Next, in order to incorporate graphing components into map layouts, we’ll use the Data Plotly plugin. This is one of the new features within QGIS 3.x. To install plugins, utilize the Plugins Main Menu:

QGIS Plugins

  • Search for DataPlotly. When the plugin loads, choose to Install plugin:

DataPlotly

  • Once installed, the following icon will be placed on the main Toolbar menu. Go ahead and click atop the icon to open the DataPlotly toolset:

DataPlotly toolset

  • The first icon on the left (paintbrush and bucket) is the initial setup for graph. Here we choose a bar graph, the layer and the x and y axis. Importantly, the bar color, stroke color and width can be determined:

Parameters and Properties

  • Next, move to the icon below (gears) to set the title, axis labels. Once complete, click Create Plot in the lower section of the tool window. This creates a preview of the graph:

Gears Icon at DataPlotly

  • Finally, click the last icon to access the HTML expression of the chart. This code can be highlighted, copied and then placed into a text editor (do not use MS Word. In the Class 2 lab, text editors will be discussed. There are many free, accessible editors).

HTML DataPlotly Properties

  • With the code complete and copied safely to a text editor (not MS Word), Create a new layout instance, and Save Project from within Map Layout. Include the following elements:

    • Main Map elements - 311 Social Distance Request Locations + NYC Boroughs
    • Map/Graph author tag (your name) + data source
    • Map legend
    • Embedded Data Plotly graph

  • The first three elements were covered generally in Class 1, Assignment 1 so you should have some familiarity with their creation and design. The fourth element - the Data Plotly graph - is a bit more challenging, but with some practice both valuable and efficient.

  • To start the graph, click the Add a New HTML to the Layout icon at left and draw out a element box on the Layout Canvas:

Add copied HTML to Layout

  • The result is a new empty element both on the map and in the right items panel. Copy the HTML code from the text editor into the HTML source section with the Source button activated. Click Return and the chart will appear on the Layout Canvas:

New HTML element in Layout Canvas

Tips:

Actions - Horizontal Properties Positioning

Video Guides:

A Missing CRS

Area & Rates

Assignment References:

Additional References:

Additional Class 2 Spatial realated data/readings/articles:

To follow, a short tutorial on Degree|Minute|Second conversion to Decimal lat/lon expression:

latlonDMS

Further, you can make the DMS>Decimal Degrees and vice versa on a point-by-point basis using an online converter like the following:

Additional ‘out of the box’, downloadable Lat/Lon datasets: